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Program Description 1 

Read Naturally is a supplemental reading program designed to 
improve reading fluency, accuracy, and comprehension of elementary 
and middle school students using a combination of books, audio 
CDs, and computer software. The program combines three main 
strategies: modeling of story reading, repeated reading of text for 
developing oral reading fluency, and systematic monitoring of student 
progress by teachers and the students themselves. Students work 
at a reading level appropriate for their achievement level, progress 
through the program at their own rate, and work (for the most part) on 
an independent basis. The program has two versions: one in which 
students use audio CDs with hard-copy reading materials, and one in 
which students use the Read Naturally computer program alone. 

Research 

The What Works Clearinghouse (WWC) identified one study of Read 
Naturally that both falls within the scope of the Adolescent Literacy 
topic area and meets WWC evidence standards. This study meets 
standards with reservations and included 156 adolescent readers in 
grades 3-5 in one location. 2 

The WWC considers the extent of evidence for Read Naturally to be small for the outcome domain of general 
literacy achievement. There were no studies that meet standards in the three other domains, so we do not report 
on the effectiveness of Read Naturally for those domains in this report. (See the Effectiveness Summary on p. 4 for 
further description of all domains.) 

Effectiveness 

Read Naturally was found to have potentially positive effects on general literacy achievement for adolescent readers. 


Table 1. Summary of findings 3 




Improvement index 

(percentile points) 




Outcome domain 

Rating of 
effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

General literacy 
achievement 

Potentially 
positive effects 

+10 

na 

1 

156 

Small 


na = not applicable 
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Program Information 

Background 

Developed by Candyce Ihnot, Read Naturally is distributed by Read Naturally, Inc., 2945 Lone Oak Drive, Suite 190, 
Saint Paul, MN 55121. Email: info@readnaturally.com. Web: www.readnaturally.com. Telephone: (651) 425-4058 or 
(800) 788-4085. Fax: (651) 452-9204. 

Program details 

Read Naturally is a reading program that is designed to supplement a school’s core language arts instruction. The pro- 
gram, which aims to improve fluency, accuracy, and comprehension by increasing the time students spend reading, is 
typically used during class time, as a pull-out intervention during the school day, or as part of an after-school program. 
The program utilizes three main strategies: (a) modeling of story reading, (b) repeated reading of text for developing 
oral reading fluency, and (c) progress monitoring. Implementing the program involves the following steps: 

(a) To help students model correct pronunciation, rate, and expression, students listen to, and read along with, a 
recording of a fluent reader reading a story; 

(b) Students engage in 1 -minute practice readings to build their mastery of the passage. Once students feel they 
can achieve their reading speed goal, they alert the teacher. The teacher then conducts a “pass timing” during 
which students are evaluated against four criteria: (1) student reaches goal rate, (2) student makes three or fewer 
errors, (3) passage is read with appropriate phrasing, and (4) comprehension questions are answered correctly. If 
students don’t meet these criteria, they spend additional time practicing the reading of the passage and then the 
teacher conducts the “pass timing” again. 

(c) Students graph their scores to track their progress from the initial reading to the final reading of each story. 

The graphs also show students’ progress over successive stories. These tools aim to ensure teacher and student 
awareness of each student’s progress. 

There are two ways that Read Naturally products can be used. The first involves using audio CDs in conjunction 
with hard-copy reading materials (Read Naturally Encore; Read Naturally Masters Edition) and the second uses the 
computer program (Read Naturally-Software Edition; Read Live). Read Naturally-Software Edition is a stand-alone 
or school network version, while Read Live is an online version of the computer program. 

The Read Naturally materials come with a teacher’s manual that includes the rationale for the program, descriptions 
of materials needed to implement the program, instructions for implementing the program, and lesson plans for 
introducing the program to students. 


Cost 

Individual Read Naturally materials vary in price. The audio CDs cost $129. The computer program costs $125 per 
reading level for one computer and $399 per reading level for a school network version. Read Live, the online soft- 
ware version, is priced per seat ranging from $149 for one seat to $1 ,999 for 130 seats. Teacher training is available 
at additional cost. Additional materials, including timers, posters, glossaries, crossword puzzles, and assessment 
materials, are also available. 
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Research Summary 

The WWC identified 56 studies that investigated the effects of Read 
Naturally on the literacy skills of adolescent readers. The WWC 
reviewed four of those studies against group design evidence stan- 
dards. One study (Heistad, 2008) is a quasi-experimental design 
that meets WWC evidence standards with reservations. The study is 
summarized in this report. Three studies do not meet WWC evidence 
standards. The remaining 52 studies do not meet WWC eligibility screens 

for review in this topic area. Citations for all 56 studies are in the References section, which begins on p. 5. 


Table 2. Scope of reviewed research 


Grade 

3,4,5 

Delivery method 

Individual/Small group 

Program type 

Supplement 


Summary of studies meeting WWC evidence standards without reservations 

No studies of Read Naturally met WWC evidence standards without reservations. 

Summary of study meeting WWC evidence standards with reservations 

Heistad (2008) examined the effects of Read Naturally on the reading achievement of students in grades 3-5 who 
were enrolled in elementary schools in the Minneapolis Public School district. Students in four elementary schools 
that were implementing Read Naturally were matched with comparison students from other schools in the same 
district based on pretest score, grade, demographic variables, and the Adequate Yearly Progress (AYP) status of 
their school. Read Naturally was implemented as a supplemental reading intervention with individual and small 
groups of students. Two schools implemented Read Naturally as a pull-out intervention during the school day, 4 
while two schools used it as part of an after-school program. Students in the comparison group attended schools 
that were not implementing Read Naturally. A total of 156 students were included in the study’s analysis, with 78 
students in each of the intervention and comparison groups. 
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Effectiveness Summary 

The WWC review of Read Naturally for the Adolescent Literacy topic area includes student outcomes in four 
domains: alphabetics, reading fluency, comprehension, and general literacy achievement. The one study of Read 
Naturally that met WWC evidence standards reported findings in one of the four domains: general literacy achieve- 
ment. The findings below present the authors’ estimates and WWC-calculated estimates of the size and statistical 
significance of the effects of Read Naturally on adolescent readers. For a more detailed description of the rating of 
effectiveness and extent of evidence criteria, see the WWC Rating Criteria on p. 16. 

Summary of effectiveness for the general literacy achievement domain 

One study that met WWC standards with reservations reported findings in the general literacy achievement domain. 

Heistad (2008) reported, and the WWC confirmed, a statistically significant positive effect for the reading portion of 
the Northwest Achievement Levels Test (NALT). The WWC characterizes this study finding as a statistically signifi- 
cant positive effect. 

Thus, for the general literacy achievement domain, one study showed a statistically significant positive effect. This 
results in a rating of potentially positive effects, with a small extent of evidence. 


Table 3. Rating of effectiveness and extent of evidence for the general literacy achievement domain 


Rating of effectiveness 

Criteria met 

Potentially positive effects 

Evidence of a positive effect 
with no overriding contrary 
evidence. 

In the study that reported findings, the estimated impact of the intervention on outcomes in the general literacy 
achievement domain was positive and statistically significant. 

Extent of evidence 

Criteria met 

Small 

One study that included 156 students in at least four schools reported evidence of effectiveness in the general 
literacy achievement domain. The number of comparison schools was not reported in the study, so the exact 
number of schools included in the analysis cannot be determined. 
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Appendix A: Research details for Heistad (2008) 

Heistad, D. (2008). The effects of Read Naturally on fluency and reading comprehension: A supplemen- 
tal service intervention (four-school study) (Unpublished manuscript). 

Table A. Summary of findings Meets WWC evidence standards with reservations 


Study findings 
Average improvement index 

Outcome domain Sample size (percentile points) Statistically significant 


General literacy achievement 156 students +10 Yes 


Setting 

The study took place in the Minneapolis Public School district, in schools that were not on the 


No Child Left Behind list of schools failing to make adequate yearly progress in 2003. 

Study sample 

Read Naturally was implemented with students in grades 3-5 in four elementary schools in the 
Minneapolis Public School district. Comparison group students were drawn from the same 
grade in the same school district. The author does not specify the number of schools attended 
by comparison group students. Students were selected for the Read Naturally intervention 
based on parent and teacher recommendations and, according to the author, were generally 
not considered to be “on course” for proficiency on the state assessments administered in the 
spring of grades 3-5. The analysis sample included 156 students in grades 3-5 (78 in Read 
Naturally and 78 in the comparison group); 46 in grade 3, 66 in grade 4, and 44 in grade 5. The 
demographic characteristics of the Read Naturally students were: 56% male, 12% classified 
as special education, 35% classified as English language learners (ELL), and 65% receiving 
free or reduced-price lunch. With respect to race and ethnicity, 39% of the intervention group 
students were Hispanic, 35% were African American, 22% were White, and 4% were Native 
American. No similar demographic information for the comparison sample was presented in 
the study. 

Intervention 

group 

Two schools used the Read Naturally Masters version that employed audio cassettes and 
hard-copy reading materials, while two schools used the Read Naturally-Software Edition. Two 
schools implemented Read Naturally as a pull-out intervention during the school day, while 
two schools used it as part of an after-school program. No further information was provided in 
the study regarding the intervention condition. 

Comparison 

group 

The study author created a matched comparison group from within the Minneapolis Public 
School district using students that were not receiving the Read Naturally program. Students 
were first matched by a pretest score on the Northwest Achievement Levels Test (NALT)- 
Reading measure, followed by the following demographic factors: grade, ELL status, special 
education status, free or reduced-price lunch status, race/ethnicity, home language, and gen- 
der. Read Naturally students were only matched to other students attending schools classified 
with the same AYP status as their own school. 
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Outcomes and 
measurement 


Support for 
implementation 


The study included one eligible outcome measure, the reading portion of the NALT, a state- 
based adaptive assessment. The NALT is administered in the spring, with prior year’s NALT 
scores used as a pretest measure in the study. For a more detailed description of this outcome 
measure, see Appendix B. The reading portion of the Minnesota Comprehensive Assessment 
(MCA) was also administered to the subsample of 88 students in grades 3 and 5 at posttest in 
the spring of 2004. However, the results for the subsample of students using this outcome mea- 
sure are not included in this review because baseline equivalence for the analysis sample was 
not established. 

One teacher in each intervention group school was trained in Read Naturally procedures by a 
Read Naturally instructor. Training included: initial assessment of student level of instruction 
using curriculum-based measurement procedures, placement procedures, use of comprehen- 
sion assessments and strategies, student goal setting, and progress monitoring procedures. 
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Appendix B: Outcome measure for the general literacy achievement domain 


General literacy achievement 


Northwest Achievement Levels 
Test (NALT)-Reading 


The NALT-Reading is a multiple-choice, standardized test aligned with state reading standards. The NALT is an 
adaptive assessment, where the version of the test taken by the student is based on their reading achievement 
level as determined by prior assessment (as cited in Heistad, 2008). 
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Appendix C: Findings included in the rating for the general literacy achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Heistad, 2008 a 

Northwest Achievement 
Levels Test (NALT)-Reading 

Grades 

3-5 

156 students 

195.40 

(9.22) 

192.90 

(11.23) 

2.51 

0.24 

+10 

0.02 


Domain average for general literacy achievement (Heistad, 2008) 0.24 +10 Statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the average change expected for all students 
who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average student's percentile rank that can be expected if the student is given the intervention. 

a For Heistad (2008), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported in the 
original study. Note that, according to WWC standards, the Adolescent Literacy team computed effects by calculating a pooled standard deviation using individual standard deviations 
for each treatment arm in Table 3 in the study, as opposed to the pooled standard deviation from a paired sample t - test in Table 4. This study is characterized as having a statisti- 
cally significant positive effect because, for a single outcome measure, the effect is positive and statistically significant. For more information, please refer to the WWC Standards and 
Procedures Handbook, version 2.1 , p. 96. 
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Appendix D: Description of supplemental findings for the general literacy achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Heistad, 2008b a 

Northwest Achievement 
Levels Test (NALT) — Reading 

Grades 

3-5 

102 students 

195.80 

(8.42) 

193.40 

(11.72) 

2.40 

0.23 

+9 

0.09 


Table Notes: The supplemental findings presented in this table are subgroup findings from the study in this report that do not factor into the determination of the intervention 
rating. This subgroup consists of students that used Read Naturally as a pull-out intervention during the school day, along with their matched comparison group, which is a 
subsample of the full study analysis reported in Heistad (2008). For mean difference, effect size, and improvement index values reported in the table, a positive number favors the 
intervention group and a negative number favors the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, represent- 
ing the average change expected for all students who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate 
presentation of the effect size, reflecting the change in an average student's percentile rank that can be expected if the student is given the intervention. 
a For Heistad (2008b), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported in 
the original study. 
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Endnotes 

1 The descriptive information for this program was obtained from a publicly available source: the program’s website (www. 
readnaturally.com; downloaded September 2012). The WWC requests developers review the program description sections for 
accuracy from their perspective. The program description was provided to the developer in October 2012 and we incorporated 
feedback from the developer. Further verification of the accuracy of the descriptive information for this program is beyond the scope 
of this review. The literature search reflects documents publicly available by April 2012. 

2 At the WWC’s request, the author provided separate results for only the subsample of students in third grade (Heistad, 2008c), which 
are reported separately in the WWC Beginning Reading intervention report. 

3 For criteria used in the determination of the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 1 6. These 
improvement index numbers show the average and range of student-level improvement indices for all findings across the studies. 

4 The additional citation of Fleistad (2008b) presents outcome data only for the subset of students attending schools that used Read 
Naturally as a pull-out intervention during the school day, along with their matched comparison group. These results are presented in 
Appendix D and do not factor into the determination of the intervention rating. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, March). 
Adolescent Literacy intervention report: Read Naturally. Retrieved from http://whatworks.ed.gov. 
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WWC Rating Criteria 

Criteria used to determine the rating of a study 


Study rating 

Criteria 

Meets WWC evidence standards 
without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 

Meets WWC evidence standards 
with reservations 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 
attrition that has established equivalence of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statistically significant positive effects, at least one of which met WWC evidence 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 

of studies show indeterminate effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC evidence 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 
The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 
The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 

Single-case design 
Standard deviation 


Statistical significance 


Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent 
of evidence levels are given in the WWC Rating Criteria on p. 16. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Rating Criteria on p. 16. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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